Determining driver engagement with autonomous vehicle

ABSTRACT

A method, system, and computer program product of controlling driver interaction with an autonomous vehicle (AV) system for a vehicle are provided. In an embodiment, a signal indicating a present state of the driver is received. A signal indicating a past state of the driver is received. A present effectiveness of the driver is determined based on the received signals. A target level of engagement of the driver with the AV system is determined based on the present effectiveness of the driver.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 15/222,516 entitled DETERMINING DRIVER ENGAGEMENT WITHAUTONOMOUS VEHICLE filed Jul. 28, 2016 which is incorporated herein byreference for all purposes.

BACKGROUND OF THE INVENTION

The introduction of an autonomous vehicle system (AV) into a vehicleposes challenges for drivers operating the vehicle and for monitoringthe driver's behavior. The AV can control the vehicle to variousdegrees, for example braking, accelerating, and/or steering. The AV maybe engaged and disengaged such that the vehicle transitions betweencontrol by the AV system and by the driver.

When an AV is available for a vehicle, there are several challenges fordrivers and monitoring the driver's behavior, including determining thedriver's readiness to operate the vehicle, monitoring driver, andmaintaining driver alertness.

It is important to accurately determine when a driver is ready tooperate the vehicle with the aid of the AV because if an AV isde-activated prematurely driving accidents may result because a driveris confused or unable to share control of the vehicle with the AV. Whenan AV system is engaged for at least part of a trip, it may be moredifficult to monitor the driver and maintain driver alertness comparedto monitoring the driver without an active AV. Conventional methods formonitoring a driver include a percentage of eye closure system (PERCLOS)and steering wheel sensors, which measure eyelid closure, eye gaze,fitness to lane to gauge the drowsiness of a driver. Conventionalmethods for maintaining driver alertness include requiring a driver toperiodically push a button inside the vehicle. However, theseconventional methods do not effectively monitor or maintain driveralertness. Thus, there is a need in the art to more effectively monitorand maintain driver alertness.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a block diagram illustrating an embodiment of a systemincluding a vehicle event recorder.

FIG. 2 is a block diagram illustrating an embodiment of a vehicle eventrecorder.

FIG. 3 is a block diagram illustrating an embodiment of a vehicle dataserver.

FIG. 4 is a flowchart illustrating an embodiment of a process fordetermining driver engagement with an autonomous vehicle system.

FIG. 5 is a flowchart illustrating an embodiment of a process fordetermining driver engagement with an autonomous vehicle system.

FIG. 6 is a flowchart illustrating an embodiment of a process forpracticing a driver skill.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

Techniques of the present disclosure determine driver readiness to takecontrol of the vehicle, monitor driver behavior, and maintain driverskill. In particular, the techniques described herein find applicationin vehicles with an autonomous vehicle system (AV), where the AV cancontrol vehicle operation, for example braking, accelerating, and/orsteering.

When an AV system is engaged for at least part of a trip, a driver mightnot always be ready to operate the vehicle with an active AV. Over time,a driver's readiness to take control from the AV may also change. Driverreadiness can be based on the driver's ability to take control of thevehicle when the vehicle transitions between engagement anddisengagement of the AV. For example, a driver who has difficulty takingcontrol of the vehicle after a period of idleness is not ready tooperate a vehicle having an AV. Whether a driver has difficulty takingcontrol of the vehicle can be determined by monitoring driver and/orvehicle behavior, as further described herein. Conventional techniquesdoes not adequately help determine driver readiness.

When an AV system is engaged for at least part of the trip, it may bemore difficult to monitor a driver because when an AV controls thevehicle, vehicle behavior does not reflect on the driver's skill in thesame way as monitored driver behavior for a vehicle without an activeAV. For example, vehicle behavior may be due at least on part on the AV.Conventional techniques do not adequately help determine driverreadiness, for example, because they do not identify what activity adriver is doing and whether the activity decreases the driver'seffectiveness.

When an AV system is engaged for at least part of a trip, a driver'sskill may change differently over time compared with traditional drivingsituations not involving AVs. For example, a driving skill may atrophywhen the AV is controlling the vehicle and the driver does not have anopportunity to practice the skill. Conventional techniques do notadequately help maintain driver alertness, for example, because they donot target a particular driving skill that degrades due to lack ofpractice.

FIG. 1 is a block diagram illustrating an embodiment of a system 100 forcontrolling driver interaction with an autonomous vehicle system. Thesystem 100 includes a processor 102, an autonomous vehicle system (AV)106, a vehicle data server 108, sensors 104, and an advanced driverassistance system (ADAS) 116.

The processor 102 is configured to perform the methods described herein.For example, the processor 102 monitors a driver's skill and changes tothe driver's skill over time to determine the driver's effectiveness,experience, and coaching needs. The processor 102 establishes a newbaseline skill level if the processor observes that the driverconsistently performs at a particular level. The processor 102 measuresthe change in a driver's skill over time by comparing the driver'sperformance with previous events and trips. The processor 102 determineswhen a driver is capable of taking control of a vehicle having an AV, asfurther described herein with respect to FIGS. 4 and 5. The processor102 is configured to determine when to engage and disengage the AVduring a trip according to the techniques described herein. Theprocessor 102 maintains driver alertness. For example, even when the AVmode is available, the processor 102 can recommend that the driver takecontrol to practice a driving skill, as further described herein withrespect to FIG. 6. In some embodiments, the processor 102 is configuredto record events and report the events to the vehicle data server 108.

In an embodiment, the processor 102 includes a vehicle event recorder(not shown). In an alternative embodiment, the vehicle event recorder isexternal to the processor 102 and coupled to the processor, using atleast part of the processing function of the processor to perform thefunctions of the vehicle event recorder described herein. The vehicleevent recorder stores data about the vehicle such as data collected bythe sensors 104. In some embodiments, the vehicle event record analyzesdata collected by the sensors 104 to identify and determine “events,”and record these events. An event can be vehicle sensor readingsindicating a particular set of circumstances such as risky drivingbehavior, arrival at a location, road conditions, and the like. An eventor set of circumstances can be determined based on past experience, apattern of sensor readings, matching sensor readings to predeterminedthreshold values, etc. The vehicle event recorder is further describedherein, e.g., with respect to FIG. 2.

The processor 102 is communicatively coupled to the AV 106. Theprocessor 102 is communicatively coupled to the vehicle data server 108over a network 112. In the example shown, the processor 102 is onboard avehicle. In an alternative embodiment, the processor 102 is providedoutside the vehicle and communicatively coupled to the vehicle. Forexample, at least some of the functions of the processor 102 can beimplemented in the vehicle data server 108. In some embodiments, theprocessor is coupled to a data storage or include a data storage, thedata storage storing instructions executable by the processor to performthe methods described herein.

The AV 106 is configured to control a vehicle. For example, the AVcontrols the brakes, acceleration, and/or steering of the vehicle. Whenthe AV is in control of the vehicle, the driver may relinquish controlof the vehicle. The driver may monitor and engage with the AV to varyingdegrees. In some embodiments, the driver may monitor the AV by placinghands on the steering wheel, which indicates preparedness to take overcontrol of the vehicle. In other embodiments, the driver may monitor theAV with hands off the steering wheel and receive a warning to prepare totake over control of the vehicle.

The sensors 104 include recorders that capture visual and audio aspectsof the vehicle. Example sensors include a video recorder, an audiorecorder, an accelerometer, a gyroscope, a vehicle state sensor, aproximity sensor, a global positioning system (GPS), a temperaturesensor, moisture sensor, a laser line tracker sensor, and the like. Avideo recorder can include an exterior video camera, an exterior stillcamera, an interior video camera, an interior still camera. An audiorecorder can include a microphone. The sensors 104 can record a state ofthe vehicle. Example sensors include a speedometer, an accelerator pedalsensor, a brake pedal sensor, an engine revolutions per minute (RPM)sensor, an engine temperature sensor, a headlight sensor, an airbagdeployment sensor, driver and passenger seat weight sensors, ananti-locking brake sensor, an engine exhaust sensor, a gear positionsensor, a cabin equipment operation sensor, and the like.

The ADAS 116 includes systems for driving safety such as rear viewcameras to assist in backing up the vehicle, lane departure detectors,and any other appropriate system. The ADAS 116 provides informationregarding driving behavior that can be useful for determining a driver'seffectiveness. In some embodiments, one or more of the sensors 104 andADAS 116 are integrated with the processor 102.

The vehicle data server 108 stores state data including driver data andvehicle data. The state data can include various states recorded overtime. In an embodiment, the vehicle data server 108 stores informationsuch as policy information and/or environmental information. Policyinformation includes policies defined by organizations that can beapplied across one or more driving situations. For example, policy candictate that AV is disengaged whenever it is raining regardless ofdriver effectiveness. Environmental information includes data regardingdriving conditions such as constructions zones, heavy traffic areas, andthe like. In an embodiment, the vehicle data server 108 is external toor remote from the vehicle. For example, the vehicle data server 108 canbe located at a cloud service provider, a home station (e.g., a shippingcompany office, a taxi dispatcher, a truck depot, etc.), a colocationcenter (e.g., a center where equipment, space, and bandwidth areavailable for rental), and the like. The vehicle data server 108 isconfigured to: receive event data and driver behavior data detected bythe sensors 104, receive data from one or more vehicle event recorders,analyze data such as data received from the sensors 104, display datasuch as the data from the sensors 104 and/or analyzed data. The datarecorded by the sensors 104 (“sensor data”) can be downloaded to thevehicle data server 108 in various ways. For example, the sensor datacan be downloaded when the vehicle reaches the home station, downloadedwirelessly, etc. The vehicle data server 108 is further describedherein, e.g., with respect to FIG. 3.

The network 112 includes any combination of: a wireless network, a wirednetwork, a cellular network, a Code Division Multiple Access (CDMA)network, a Global System for Mobile Communication (GSM) network, aLong-Term Evolution (LTE) network, a Universal Mobile TelecommunicationsSystem (UMTS) network, a Worldwide Interoperability for Microwave Access(WiMAX) network, a Dedicated Short-Range Communications (DSRC) network,a local area network, a wide area network, the Internet, or any otherappropriate network.

In operation, the processor 102 receives present state data from thesensors 104. The processor 102 receives past state data from the vehicledata server 108. The processor 102 provides an output to the driver 114via a user interface. Based on the present state data and the past statedata, the processor 102 determines a target engagement level of thedriver and the AV according to the methods described herein. Theprocessor 102 then provides a recommendation to the driver 114 to adjustthe AV or a warning to the driver 114 that the AV 106 will be engaged ordisengaged.

FIG. 2 is a block diagram illustrating an embodiment of a vehicle eventrecorder 200. The vehicle event recorder 200 includes a processor 202,data storage 204, one or more communications interfaces 206, and asensor interface 208.

The processor 202 controls operations of the vehicle event recorder 200such as reading from and writing to the data storage 204, communicatingvia the communications interface 206, and/or reading data via the sensorinterface 208. In an embodiment, the processor 202 can store a set ofcircumstances as an event. The processor can determine whether to storean event based on sensor readings indicating a particular set ofcircumstances such as risky driving behavior, arrival at a location,road conditions, and the like. An event or set of circumstances can bedetermined based on past experience, a pattern of sensor readings,matching sensor readings to predetermined threshold values, etc. Forexample, the processor receives compliance data, where the compliancedata describes sensor readings matching an acceptable behavior. Theprocessor compares sensor readings to the compliance data. In anembodiment, if the sensor readings do not meet the compliance data, theprocessor stores the sensor readings as a non-compliant event. In anembodiment, if the sensor readings meets the compliance data, theprocessor stores the sensor readings as a compliant event or does notstore the sensor readings as an event.

The data storage 204 stores instructions executable by the processor 202to perform functions of the vehicle event recorder. The data storage 204stores data about the driver and one or more vehicles such as scores fordriver readiness, present state data, vehicle event recorder data,sensor data, video data, map data, machine learning algorithm data, orany other appropriate data. The data storage 204 can be implementedusing one or more of: random access memory (RAM), read only memory(ROM), nonvolatile memory, flash memory, a hard disk, and the like.

The communications interface(s) 206 can include one or more device forinput and output from the vehicle event recorder. The communicationsinterface(s) can enable communication with any type of network, e.g.,packet-switched and/or circuit-switched networks. The communicationsinterface(s) can include any combination of a wired interface or awireless interface. Example communication interfaces include a GSMinterface, a CDMA interface, a LTE interface, a Wi-Fi® interface, anEthernet interface, a Universal Serial Bus (USB) interface, a Bluetooth®interface, an Internet interface, and the like.

The sensor interface 208 includes an interface to one or more sensorssuch as the sensors 104 shown in FIG. 1. In an embodiment, the sensorinterface 208 includes an on-board diagnostics (OBD) bus (e.g., societyof automotive engineers (SAE) J1939, J1708/J1587, OBD-II, CAN BUS,etc.). The vehicle event recorder 200 can communicate with the sensorsvia the OBD bus, wireless connection, and the like.

FIG. 3 is a block diagram illustrating an embodiment of a vehicle dataserver. The vehicle data server 300 includes a processor 302, datastorage 304, and communications interface(s) 306.

The processor 302 controls operations of the vehicle data server 300such as the methods described herein and/or determining a route, set ofroute segments, a route segment safety score, a speed distribution,collecting speed data, and the like. The processor 302 receives datafrom one or more vehicle event recorders, and determines driverengagement with the AV based on the received data and past data state(stored in the data storage 304) according to the methods describedherein.

The data storage 304 stores instructions executable by the processor 302to perform functions of the vehicle data server. The data storage 304stores past state data, vehicle event recorder data, sensor data, videodata, map data, machine learning algorithm data, or any otherappropriate data. The data storage 304 can be implemented using one ormore of: RAM, ROM, nonvolatile memory, flash memory, a hard disk, andthe like.

The communications interface(s) 306 can include one or more device forinput and output from the vehicle event recorder. The communicationsinterface(s) can enable communication with any type of network, e.g.,packet-switched and/or circuit-switched networks. The communicationsinterface(s) can include any combination of a wired interface or awireless interface. Example communication interfaces include a GSMinterface, a CDMA interface, a LTE interface, a Wi-Fi® interface, anEthernet interface, a USB interface, a Bluetooth® interface, an Internetinterface, and the like.

FIG. 4 is a flow diagram illustrating an embodiment of a method 400 forcontrolling driver interaction with an autonomous vehicle system (AV).The method 400 can be performed by a processor (e.g., the processor 102and/or the vehicle data server 108 of FIG. 1). In 402, the method 400receives present state data. The present state data includes data abouta current state of the driver such as reactions to the environment,alertness, emotion, and the like. The present state data can include orbe based on one or more sensor readings. Present state data about adriver can include an evaluation of the driver's performance. Forexample, a sensor reading can be compared with a target or thresholdvalue to determine deviation from an expected value. The deviation canindicate the driver's performance such as the driver's effectiveness.Example sensors include video recorders and audio recorders. A videorecorder with 4D imaging capabilities determines what the driver iscurrently doing based on detection of the position and movement of thedriver's body. The video recorder can detect an activity that a driveris engaged in such as reading a book, texting, sleeping, and the like.The video recorder can detect eye movement to determine a driver'sstate. For example, frequent eye blinking indicates that the driver islikely to be drowsy. The audio recorder can detect sounds inside thevehicle. For example, a yawn indicates that the driver is likely to bedrowsy. As further described herein, other sensors and combinations ofsensors are possible.

In 404, the method 400 receives past state data. The past state dataincludes data about the vehicle and a past state of the driver. Thevehicle data includes information about the vehicle such as make, model,and maintenance history. The past state information can be stored as ananalytical model associated with an identity of the driver. The vehicledata can affect driver behavior and be used to predict driver behavior.The past state of the driver can include events such as how recently thedriver napped, the duration of a previous work shift, and the like. Thepast state of the driver can be captured in a behavior profile. Thebehavior profile includes an identification of the driver and thedriver's behavioral patterns. The past behavior of the driver canpredict future driving behavior. For example, a driver will likely reactto a future situation in a similar way to how the driver reacted tosimilar situations in the past. The driver's past performance andinteraction with an AV can also predict future interactions with the AV.Thus, the identification of the driver (e.g., a corresponding behaviorprofile) indicates a general effectiveness or experience of the driverand establishes expectations for driver behavior and reactions to roadconditions.

In 406, the method 400 determines the driver's effectiveness based onthe present and past state data. The driver's effectiveness isdetermined by the present state data received in 402. For example, thedriver's effectiveness is directly related to the alertness of thedriver and inversely related to the driver's drowsiness. The driver'seffectiveness is also determined by the past state data received in 404.For example, how easily the driver gets drowsy or distracted affects hiseffectiveness. As another example, the driver's age, experience with aparticular type of vehicle, physical fitness, previous trafficviolations, and the like affects his effectiveness. Certain past eventssuch as a suspended driver's license can disqualify the driver fromdriving. The driver's pattern of behavior can also indicate the driver'seffectiveness over the course of a trip. As further described herein,the driver's past behavior can predict future behavior. Thus, a driver'spast pattern of behavior can indicate his effectiveness. The driver'seffectiveness can change during the course of a trip. For example, adriver who, in the past, has exhibited decreased effectiveness after sixhours of driving is expected to have decreased effectiveness after sixhours on a current trip. The method 400 can score the driver'seffectiveness. The driver's effectiveness score can be determined basedon a weighting of factors including a weighting of present and paststate data.

In 408, the method 400 determines a target engagement level of thedriver with the AV based on the effectiveness of the driver. Forinstance, a relatively effective driver has a higher target engagementlevel because an effective driver is better prepared to take overcontrol of the vehicle and transition between engagement anddisengagement of the AV compared with a less effective driver. On theother hand, a relatively ineffective driver has a lower targetengagement level because an ineffective driver has a more difficult timetaking control of the vehicle and transitioning between engagement anddisengagement of the AV.

In an alternative embodiment, the target engagement level is based onhow attentive the driver is and whether the attentiveness of the driveris sufficient to maintain an experience rating. The attentiveness of thedriver can be determined based on driver reaction to road conditions.For example, the driver's level of drowsiness (e.g., measured by aninterior video recorder) can lower the attentiveness of the driver.Better vehicle handling indicates greater driver attentiveness. Thequality of vehicle handling can be based on whether the vehicle staysbetween lane markers, how close a vehicle wheel is to a lane marker,maintaining distance between vehicles in front of and behind thevehicle, handling adverse conditions, and the like. An experience ratingcan be based on one or more past states of the driver. For example, adriver who is typically highly attentive can be given a higherexperience rating than one who is typically not as highly attentive. Asanother example, a driver who has experienced broader conditions (e.g.,road types, weather, traffic, etc.) and/or handled a vehicle for alonger period of time (e.g., taking a longer shift, driving for moreyears) has a higher experience rating than one who has not had recentexperiences or handled the vehicle for as long.

The target engagement level can be determined dynamically as the targetengagement level may change with a driver's changing effectiveness. Forexample, a driver who is historically ineffective after being idle forsix hours may have a relatively high target engagement level at thebeginning of the six-hour period and a relatively low target engagementlevel at the end of the six-hour period. The target engagement level candefine, among other things, (i) how frequently the AV is engaged whenthe driver is inside the vehicle, (ii) driving situations in which theAV should be engaged, and (iii) whether or not a driver should be incontrol of the vehicle. Whether a driver should be in control of avehicle may be due to various factors such as a need to practice adriving skill or the driver's readiness to operate a vehicle having anAV system. In an embodiment, one or more of the factors can be moreheavily weighted or override one or more of the other factors in makingthe determination of whether the driver should be in control of thevehicle.

In 412, the method 400 determines whether the present engagement levelof the driver with the AV is higher than, lower than, or the same as thetarget engagement level. The present engagement level of the driver withthe AV is a measure of how engaged the driver is with the AV. Thedriver's engagement with the AV is high if the AV is engaged andcontrolling the vehicle. The driver's engagement with the AV is low ifthe AV is disengaged and not controlling the vehicle.

In 412, if the method 400 determines that the present engagement levelis higher than the target engagement level, the method 400 proceeds to414, in which the method makes a recommendation to disengage the AVsystem. The recommendation may be in the form of an audio signal, avisual signal, a haptic signal, or the like to prompt the driver todisengage the AV system. In an alternative embodiment, the method 400automatically disengages the AV system. In some embodiments, when the AVsystem is disengaged, the driver takes over control of the vehicle andhas an opportunity to practice a driving skill. This helps a driver tomaintain or improve the driving skill. In some embodiments, when the AVsystem is disengaged, the vehicle will be automatically guided to astop. This improves safety by preventing the driver from operating thevehicle when the driver is unfit. As further described herein, in 414,disengaging the AV may include providing a warning to the driver priorto the disengagement, the timing of the warning being dependent on thedriver's effectiveness determined in 406.

In 412, if the method 400 determines that the present engagement levelis the same as the target engagement level, the method 400 proceeds to416, in which the AV system remains unchanged.

In 412, if the method 400 determines that the present engagement levelis lower than the target engagement level, the method 400 proceeds to418, in which the method makes a recommendation to engage the AV system.The recommendation may be in the form of an audio signal, a visualsignal, a haptic signal, or the like to prompt the driver to engage theAV system. In an alternative embodiment, the method 400 automaticallyengages the AV system. In some embodiments, when the AV system isengaged, the driver relinquishes control of the vehicle. This can allowthe driver to rest for a period of time, while the AV controls thevehicle. In some embodiments, the driver continues to monitor thevehicle while the AV is engaged, e.g. keeping hands on a steering wheel.As further described herein, engaging the AV may include providing awarning to the driver prior to the engagement, the timing of the warningbeing dependent on the driver's effectiveness determined in 406.

In some embodiments, the method 400 provides a notification responsiveto engagement or disengagement of the AV. The notification includesinformation about the state of the AV such as the circumstances for thechange to the AV. For example, the notification is sent over a networkto a central server (e.g., the vehicle data server 108 of FIG. 1). Insome embodiments, a dispatcher can view the notification and use theinformation in the notification to determine whether the driver needsfurther coaching or work with a simulator. In some embodiments, thedetermination of whether the driver needs further coaching or work witha simulator is performed systematically or automatically, as furtherdescribed herein.

FIG. 5 is a flow diagram illustrating an embodiment of a method 500 forcontrolling driver interaction with an AV. The method 500 can beperformed by a processor (e.g., the processor 102 and/or the vehicledata server 108 of FIG. 1).

In 502.1, the method 500 receives present state data. The present statedata includes data about a current state of the driver such as reactionsto the environment, alertness, emotion, and the like. The present statedata can include or be based on one or more sensor readings. Presentstate data about a driver can include an evaluation of the driver'sperformance. For example, a sensor reading can be compared with a targetor threshold value to determine deviation from an expected value. Thedeviation can indicate the driver's performance such as the driver'seffectiveness. Example sensors include video recorders and audiorecorders. A video recorder with 4D imaging capabilities determines whatthe driver is currently doing based on detection of the position andmovement of the driver's body. The video recorder can detect an activitythat a driver is engaged in such as reading a book, texting, sleeping,and the like. The video recorder can detect eye movement to determine adriver's state. For example, frequent eye blinking indicates that thedriver is likely to be drowsy. The audio recorder can detect soundsinside the vehicle. For example, a yawn indicates that the driver islikely to be drowsy. As further described herein, other sensors andcombinations of sensors are possible. As further described herein, thepresent data can be evaluated in accordance with the other receiveddata.

In 502.2, the method 500 receives past state data. The past state dataincludes data about the vehicle and a past state of the driver. Thevehicle data includes information about the vehicle such as make, model,and maintenance history. The past state information can be stored as ananalytical model associated with an identity of the driver. The vehicledata can affect driver behavior and be used to predict driver behavior.The past state of the driver can include events such as how recently thedriver napped, the duration of a previous work shift, and the like. Thepast state of the driver can be captured in a behavior profile. Thebehavior profile includes an identification of the driver and thedriver's behavioral patterns. The past behavior of the driver canpredict future driving behavior. For example, a driver will likely reactto a future situation in a similar way to how the driver reacted tosimilar situations in the past. The driver's past performance andinteraction with an AV can also predict future interactions with the AV.Thus, the identification of the driver (e.g., a corresponding behaviorprofile) indicates a general effectiveness or experience of the driverand establishes expectations for driver behavior and reactions to roadconditions. As further described herein, the present data can beevaluated in accordance with the other received data.

In 502.3, the method 500 receives policy data. The policy data includesdata about policies or rules that apply to a driver and/or a vehicle.The policy data can be pre-defined, for example by an administrator ordispatcher. In some instances, the policy can override at least some ofthe other received data (present state, past state, environment data).For example, the policy can provide that AV should not be engaged whenthe vehicle is in a construction zone, regardless of drivereffectiveness. As further described herein, the present data can beevaluated in accordance with the other received data.

In 502.4, the method 500 receive environment data. The environment dataincludes information about the conditions exterior to the vehicle. Forexample, the environment data include weather information (whether it israining, snowing, windy, etc.), road conditions, and the like. Theenvironment data can influence the determination of the engagement levelbecause aspects of the exterior conditions of the vehicle can affect thedriver's alertness, effectiveness, etc. For example, it can be moredifficult to control a vehicle in icy conditions. The environment data502.4 can also be used in combination with the policy data 502.3 todetermine whether one or more conditions have been fulfilling for apolicy to affect the determination of driver effectiveness, as furtherdescribed herein. As further described herein, the present data can beevaluated in accordance with the other received data.

In 504, the method 500 determines the driver's effectiveness based onthe received data, e.g., present data 502.1, past state data 502.2,policy data 502.3, and/or environment data 502.4. The driver'seffectiveness is determined by the present state data received in 502.1.For example, the driver's effectiveness is directly related to thealertness of the driver and inversely related to the driver'sdrowsiness.

The driver's effectiveness is also determined by the past state datareceived in 502.2. For example, how easily the driver gets drowsy ordistracted affects his effectiveness. As another example, the driver'sage, experience with a particular type of vehicle, physical fitness,previous traffic violations, and the like affects his effectiveness.Certain past events such as a suspended driver's license can disqualifythe driver from driving. The driver's pattern of behavior can alsoindicate the driver's effectiveness over the course of a trip. Asdescribed herein, the driver's past behavior can predict futurebehavior. Thus, a driver's past pattern of behavior can indicate hiseffectiveness.

The driver's effectiveness is also determined by the policy datareceived in 502.3. For example, the policy data can determine how toevaluate the received present state data 502.1 and the past state data502.2. The policy can increase or lower a driver effectiveness score bydefining how to treat received data. For example, the policy can bedefined as a rule, taking in received data and outputting an appropriatedriver effectiveness score. A policy can be that an AV is set to improvesafety in a rainy situation (e.g., measured rainfall exceeds athreshold). Given a same level of driver effort, the inclement weathercan render the driver less effective than in fair weather. Thus, it maybe desirable to evaluate driver effectiveness differently in somesituation, where the different evaluation is defined by the policy. Forinstance, if the AV is rudimentary and would not perform as well as thedriver in a rainy situation, the AV is disengaged. Otherwise, the AV isengaged. In other words, the AV can be set depending on an evaluation ofwhether the AV is more or less effective than the driver in a rainysituation. Accordingly to the policy, if the received data indicatesthat it is rainy, driver effectiveness can be decreased. As anotherexample, when it is rainy, the driver effectiveness score can beautomatically lowered to below a threshold, causing an AV to be set toimprove safety.

The driver's effectiveness is also determined by the environment datareceived in 502.4. For example, the environment data can determine howto evaluate the received present state data 502.1, past state data502.2, and policy data 502.3. The environment data can increase or lowera driver effectiveness score as follows. If the received data indicatesthat the road is icy, driver effectiveness can be automaticallydecreased. The amount by which the driver effectiveness is changed candepend on the identity of the driver. For example, if a driverconsistently performs 25% worse on icy roads than non-icy roads (e.g.,reflected by past state data), the driver effectiveness score can belowered accordingly when an icy condition is detected from the receivedenvironment data.

The driver's effectiveness can change during the course of a trip. Forexample, a driver who, in the past, has exhibited decreasedeffectiveness after six hours of driving is expected to have decreasedeffectiveness after six hours on a current trip. The method 500 canscore the driver's effectiveness. The driver's effectiveness score canbe determined based on a weighting of factors including a weighting ofpresent and past state data.

In 506, the method 500 determines a target engagement level of thedriver with the AV based on the effectiveness of the driver. Forinstance, a relatively effective driver has a higher target engagementlevel because an effective driver is better prepared to take overcontrol of the vehicle and transition between engagement anddisengagement of the AV compared with a less effective driver. On theother hand, a relatively ineffective driver has a lower targetengagement level because an ineffective driver has a more difficult timetaking control of the vehicle and transitioning between engagement anddisengagement of the AV. The target engagement level can be determineddynamically as the target engagement level may change with a driver'schanging effectiveness. For example, a driver who is historicallyineffective after being idle for six hours may have a relatively hightarget engagement level at the beginning of the six-hour period and arelatively low target engagement level at the end of the six-hourperiod. The target engagement level can define, among other things, (i)how frequently the AV is engaged when the driver is inside the vehicle,(ii) driving situations in which the AV should be engaged, and (iii)whether or not a driver should be in control of the vehicle. Whether adriver should be in control of a vehicle may be due to various factorssuch as a need to practice a driving skill or the driver's readiness tooperate a vehicle having an AV system. In an embodiment, one or more ofthe factors can be more heavily weighted or override one or more of theother factors in making the determination of whether the driver shouldbe in control of the vehicle.

In 508, the method 500 determines whether the present engagement levelof the driver with the AV is higher than, lower than, or the same as thetarget engagement level. The present engagement level of the driver withthe AV is a measure of how engaged the driver is with the AV. Thedriver's engagement with the AV is high if the AV is engaged andcontrolling the vehicle. The driver's engagement with the AV is low ifthe AV is disengaged and not controlling the vehicle.

In 508, if the method 500 determines that the present engagement levelis higher than the target engagement level, the method 500 proceeds to512, in which the method makes a recommendation to disengage the AVsystem. The recommendation may be in the form of an audio signal, avisual signal, a haptic signal, or the like to prompt the driver todisengage the AV system. In an alternative embodiment, the method 500automatically disengages the AV system. In some embodiments, when the AVsystem is disengaged, the driver takes over control of the vehicle andhas an opportunity to practice a driving skill. This helps a driver tomaintain or improve the driving skill. In some embodiments, when the AVsystem is disengaged, the vehicle will be automatically guided to astop. This improves safety by preventing the driver from operating thevehicle when the driver is unfit. As described herein, in 512,disengaging the AV may include providing a warning to the driver priorto the disengagement, the timing of the warning being dependent on thedriver's effectiveness determined in 504.

In 508, if the method 500 determines that the present engagement levelis the same as the target engagement level, the method 500 proceeds to514, in which the AV system remains unchanged.

In 508, if the method 500 determines that the present engagement levelis lower than the target engagement level, the method 500 proceeds to516, in which the method makes a recommendation to engage the AV system.The recommendation may be in the form of an audio signal, a visualsignal, a haptic signal, or the like to prompt the driver to engage theAV system. In an alternative embodiment, the method 500 automaticallyengages the AV system. In some embodiments, when the AV system isengaged, the driver relinquishes control of the vehicle. This can allowthe driver to rest for a period of time, while the AV controls thevehicle. In some embodiments, the driver continues to monitor thevehicle while the AV is engaged, e.g. keeping hands on a steering wheel.As further described herein, engaging the AV may include providing awarning to the driver prior to the engagement, the timing of the warningbeing dependent on the driver's effectiveness determined in 504.

In some embodiments, in 518, the method 500 provides a notificationresponsive to engagement or disengagement of the AV. The notificationincludes information about the state of the AV such as the circumstancesfor the change to the AV. For example, the notification is sent over anetwork to a central server (e.g., the vehicle data server 108 of FIG.1). In some embodiments, a dispatcher can view the notification and usethe information in the notification to determine whether the driverneeds further coaching or work with a simulator. In some embodiments,the determination of whether the driver needs further coaching or workwith a simulator is performed systematically or automatically, asfurther described herein.

The timing of providing a warning vary with the driver effectiveness(e.g., providing a warning in 414, 416 of FIG. 4 or 512, 516 of FIG. 5).For example, the timing of providing the warning can be proportional tothe driver effectiveness. As another example, if the drivereffectiveness is below a threshold, the timing can be set above awarning threshold. If the driver effectiveness is above a threshold, thetiming can be set below a warning threshold. This means that if thedriver is relatively ineffective, the driver is warned earlier than adriver who is relatively effective, giving the ineffective driver moretime to prepare for a change. The timing of the warning can be definedas a target time, where the warning is provided at or before the targettime.

FIG. 6 is a flow diagram illustrating an embodiment of a method 600 forcontrolling driver interaction with an AV. The method 600 determineswhether the driver can benefit from further coaching or work with asimulator. For instance, even where a driver may be adequately effectiveto allow the AV to take control of the vehicle, the AV can neverthelessbe disengaged to allow the driver to practice controlling the vehicle.The method 600 can be performed by a processor (e.g., the processor 102and/or the vehicle data server 108 of FIG. 1).

In 602, the method 600 determines whether a vehicle is encountering adriving situation that is relevant to a particular skill. For example, askill can be navigating a windy one-lane road. When the vehicle enters astretch of road that is a windy one-lane road, the situation would berelevant to the particular skill. The determination can be made based onreceived sensor data, present state data, and/or map data. Examples ofsensor data and present state data collection are described herein. Forexample, the sensor data is information collected by the sensors 104 inFIG. 1. As another example, the present state data is information aboutthe vehicle and/or the driver stored in the data storage 204 in FIG. 2.As another example, the map data includes data embedded in maps. Theembedded map data includes information associated with geolocationfeatures such as number of lanes, incline, speed limit, traffic controldevices (e.g., traffic lights), and the like. In some embodiments, themaps are stored in data storage 204 in FIG. 2.

If the vehicle is not encountering a driving situation relevant to aparticular skill, control returns to the start, in which the methodcontinues to monitor whether the vehicle encounters a driving situationrelevant to a particular skill.

If the vehicle is encountering a driving situation relevant to aparticular skill, control passes to 604 in which the method 600 providesa recommendation to disengage the AV. In some instances, therecommendation can be based on a current state of the AV. For example,if the AV is engaged or active, the recommendation can be a notificationto change the state of the AV to disengaged. If the AV is inactive ordisengaged, the recommendation can be a notification to maintain thestate of the AV or no notification can be made. The recommendation canprovide the driver with an opportunity to take control of the vehicleand practice a skill.

In 606, the method 600 determines whether the driver takes control. Ifthe driver takes control of the vehicle, the method can monitor thebehavior of the driver and/or the vehicle to determine whether the skillimproves in 608. The determination of whether the skill improves can bebased on collected data about the driver and/or the vehicle. Forexample, driver behavior can be tracked over time, and after the driverexceeds a threshold number of successful maneuvers, the skill can bedetermined to have improved. Using the example of navigating a windyone-lane road. A driver can be evaluated based on controlling thevehicle to stay within lane markers. When the driver stays within thelane markers for a period of time exceeding a threshold or for a numberof turns exceeding a threshold, the skill is determined to haveimproved. In an alternative embodiment (not shown), the determinationcan be of whether the skill is maintained rather than improved. Themaintenance of a skill can be determined based on collected data aboutthe driver and/or the vehicle. For example, driver behavior can betracked over time, and after the driver exceeds a threshold number ofsuccessful maneuvers, the skill can be determined to have beenmaintained. In this way, driver skill can be maintained and/or improvedfor vehicles having an AV.

If the driver's skill improves in 608, the method can terminate.Otherwise, if the driver's skill does not improve in 608, the method canprovide a recommendation for further coaching in 612. In an embodiment,the recommendation can be provided to a dispatcher, who uses thisinformation to determine whether the driver needs further coaching orwork with a simulator. In another embodiment, the recommendation can beprovided to the driver and further opportunities to practice the skillcan be provided. For example, an AV can be disengaged (or remaindisengaged) when the vehicle encounters a situation relevant to theskill.

In 606, if the driver does not take control of the vehicle, control canpass to 612, in which the method recommends coaching. The recommendationcan be provided to a dispatcher, who uses this information to determinewhether the driver needs further coaching or work with a simulator. Thefurther coaching can be performed in various other settings.

The techniques described herein find application in a variety ofsituations. For example, vehicles can be grouped into platoons (alsoknown as “platooning”) which typically involves the vehicles driving atsubstantially the same speed and with relatively small spacing betweenthe vehicles. There are numerous benefits of platooning includinggreater fuel economy, reduced congestion, shorter commutes, fewertraffic collisions, and reduced driver fatigue. However, if one or morevehicles in the platoon deviates from expected behavior, there is anincreased likelihood of accidents for other vehicles in the platoon in adomino-effect. One or more vehicles and/or drivers in a platoon can bemonitored according to the techniques described herein and platooningcan be engaged or dis-engaged based on a driver's behavior such asalertness or effectiveness.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A method comprising: receiving, by a processor, asignal indicating a present state of a driver associated with a vehicle,wherein the vehicle has an autonomous system (AV) for autonomousoperation; receiving, by the processor, a signal indicating a past stateof the driver; scoring, by the processor, effectiveness of the driverbased at least in part on the received signals; determining, by theprocessor, a target level of engagement of the driver with the AV systembased on the driver effectiveness score; determining, by the processor,whether a present driver engagement level meets the target engagementlevel; in response to a determination that the present driver engagementlevel meets the target engagement level, at least one of (i) outputtinga recommendation to the driver to disengage the AV system and (ii)automatically disengaging the AV system; and in response to adetermination that the present driver engagement level is less than thetarget engagement level, at least one of (i) outputting a recommendationto the driver to engage the AV system and (ii) automatically engagingthe AV system.
 2. The method of claim 1, wherein the target level ofengagement defines whether to recommend that the driver operate thevehicle in a non-autonomous mode.
 3. The method of claim 1, furthercomprising: receiving, by the processor, a policy, wherein thedetermination of the target level of engagement is further based on thepolicy.
 4. The method of claim 3, further comprising: at least one of(i) providing, by the processor, a recommendation to the driver and (ii)changing, by the processor, a mode of operation of the AV system, basedon a determination, by the processor, that the policy has been violated.5. The method of claim 1, wherein the determination of the target levelof engagement is further based on an attentiveness level of the driversufficient to maintain an experience rating.
 6. The method of claim 1,wherein the determination of the target level of engagement is furtherbased on an attentiveness level of the driver sufficient to operate thevehicle in an autonomous mode.
 7. The method of claim 1, furthercomprising: determining, by the processor, whether to recommend coachingfor the driver based on the effectiveness of the driver.
 8. The methodof claim 1, further comprising: determining, by the processor, whetherto recommend simulation training for the driver based on theeffectiveness of the driver.
 9. The method of claim 1, wherein thepresent state signal is based on a detection, by a sensor, of driveractivity.
 10. The method of claim 1, wherein the determination of thetarget level of engagement includes determining a target time fordisengaging the AV system.
 11. The method of claim 1, wherein thedetermination of the target level of engagement includes determining atarget time for providing a warning to the driver regardingdisengagement of the AV system.
 12. The method of claim 1, furthercomprising: determining a target time for disengaging the AV system. 13.The method of claim 1, further comprising: determining a target time forproviding a warning regarding disengagement of the AV system.
 14. Themethod of claim 1, wherein the past state signal is based at least inpart on when the driver napped.
 15. The method of claim 1, wherein thepast state signal includes at least one of: vehicle information,environment, and driver behavioral information.
 16. The method of claim1, further comprising: receiving, by the processor, environment dataindicating a driving situation; determining, by the processor, whetherthe AV system performs or is expected to perform below a threshold whenencountering the driving situation; if the AV system performs below thethreshold, at least one of: making a recommendation to the driver anddisengaging the AV system.
 17. The method of claim 16, furthercomprising: responsive to driver operation of the vehicle in anon-autonomous mode in the driving situation, determining, by theprocessor, whether a skill of the driver has improved; if the skill ofthe driver has not improved, recommending, by the processor, coachingfor the driver.
 18. The method of claim 1, further comprising:determining, by the processor, based on at least one of sensor data, mapdata and environment information, that the vehicle is encountering adriving situation relevant to a skill set of the driver; at least oneof: providing, by the processor, a recommendation to the driver tooperate the vehicle in a non-autonomous mode; disengaging, by theprocessor, the AV system.
 19. A system for controlling driverinteraction with an autonomous vehicle (AV) system for a vehicle, thesystem comprising: a processor configured to: receive a signalindicating a present state of a driver associated with an vehicle,wherein the vehicle has an autonomous system (AV) for autonomousoperation; receive a signal indicating a past state of the driver; scoreeffectiveness of the driver based at least in part on the receivedsignals; determine a target level of engagement of the driver with theAV system based on the driver effectiveness score; determine whether apresent driver engagement level meets the target engagement level; inresponse to a determination that the present driver engagement levelmeets the target engagement level, at least one of (i) output arecommendation to the driver to disengage the AV system and (ii)automatically disengage the AV system; and in response to adetermination that the present driver engagement level is less than thetarget engagement level, at least one of (i) output a recommendation tothe driver to engage the AV system and (ii) automatically engage the AVsystem; and a non-transitory memory coupled to the processor andconfigured to provide the processor with instructions.
 20. A computerprogram product for controlling driver interaction with an autonomousvehicle (AV) system for a vehicle, the computer program product beingembodied in a non-transitory computer-readable storage medium andcomprising computer instructions for: receiving, by a processor, asignal indicating a present state of a driver associated with anvehicle, wherein the vehicle has an autonomous system (AV) forautonomous operation; receiving, by the processor, a signal indicating apast state of the driver; scoring, by the processor, effectiveness ofthe driver based at least in part on the received signals; determining,by the processor, a target level of engagement of the driver with the AVsystem based on the driver effectiveness score; determining, by theprocessor, whether a present driver engagement level meets the targetengagement level; in response to a determination that the present driverengagement level meets the target engagement level, at least one of (i)outputting a recommendation to the driver to disengage the AV system and(ii) automatically disengaging the AV system; and in response to adetermination that the present driver engagement level is less than thetarget engagement level, at least one of (i) outputting a recommendationto the driver to engage the AV system and (ii) automatically engagingthe AV system.